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Abstract 

Using meta-analytic techniques this study examined the effects of applying mathematical 
modelling to support student math knowledge acquisition at the high school and college levels. 
The research encompassed experimental studies published in peer-reviewed journals between 
January 1, 2000, and February 27, 2013. Such formulated orientation called for extracting 
individual effect sizes of student achievement from the accumulated research conducting a 
moderator analysis. A systematic review of literature resulted in locating 13 primary research 
articles involving 1,670 participants. The overall mean effect size; ES = 0.69 (SE = 0.05, 95% Cl: 
0.59-0.79) of a medium magnitude and positive direction supported the claim that mathematical 
modelling helps students understand and apply math concepts. A subsequent moderator analysis 
revealed differences of the effect sizes due to different modelling designs, aim of the modelling 
process, grade levels, and content domains. The research findings along with the discussion can 
be of interest to mathematics curriculum designers and practitioners who use modelling in their 
teaching practice. 

Keywords: Mathematical modelling; Meta-analysis; Student achievement. 
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Introduction 

Mathematical modelling (MM) is defined in literature in various ways; Poliak (2007) a precursor 
of introducing MM to school practice described modelling as a process of formulating a problem 
from outside of mathematics, understanding the problem, visualizing, and solving it. Lesh and 
Harel (2003) defined MM as an activity of finding quantifiable patterns of a phenomenon and its 
generalization. A more comprehensive description of MM was proposed by Confrey and 
Maloney (2007) who stated that MM is: 

The process of encountering an indeterminate situation, problematizing it, and bringing 
inquiry, reasoning, and mathematical structures to bear to transform the situation. The 
modelling produces an outcome - a model - which is a description or a representation of 
the situation, drawn from the mathematical disciplines, in relation to the person's 
experience, which itself has changed through the modelling process (p.60). 

MM as an educational development (Poliak, 1968) was initiated in engineering and sciences 
settings, and then spread to other fields. Its purpose was to elevate the gap between reasoning in 
a mathematics class and reasoning about a situation in the real world (Blum, Galbraith, Henn, & 
Niss, 2007). Situated in contexts, MM provides methods for analyzing data, formulating 
theories—often expressed in symbolic mathematical fonns—and testing those theories as well as 
it helps with contextualizing problem solving processes. The process of MM can be exercised 
using various learning settings; from deductively arranged authentic problem modelling activities 
(e.g., English & Sriraman, 2010) to inductively organized inquiries leading the learners to 
formulating general patterns (e.g., Sokolowski & Rackly, 2011). Due to being context driven, 
knowledge acquisition by the processes of modelling plays a vital role in developing students’ 
skills not only in mathematics classes but also in other disciplines, especially in sciences (Lesh & 
Harel, 2003; Wells, Hastens, & Swackhamer, 1995). As Confrey (2007) claimed “the strongest 
arguments for modelling are based on the view that it will be advantageous for the development 
of student thinking” (p.125) which is being accomplished by shifting the learning focus from 
finding unique solutions to enhancing skills of developing general solution processes through 
transforming and interpreting information, constructing models, and validating the models (Lim, 
Tso, & Lin, 2009). Through these processes, students learn math to “develop competency in 
applying mathematics and building mathematical models for areas and purposes that are extra — 
mathematical” (Niss, Blum, & Galbraith, 2007, p.5). Concerning the underlying learning theory, 
modelling is “based upon a constructive paradigm; hence, the assumption that learning is a self- 
regulated activity which cannot be controlled from the outside but which can be encouraged at 
best” (Hussmann, 2007, p. 344). This orientation requires the teacher to guide the students 
through MM processes not provide direct solutions. 

Theoretical Background 

Lingefjard (2007) stated that “mathematical modelling is not a body of mathematical knowledge 
but rather a collection of general principles which experience has proved to be helpful in the 
process of applying mathematical know-how to analyze problems” (p. 476). As an activity 
helping students apply the concepts of mathematics outside of mathematics classroom, MM is 
characterized by a unique structure called often modelling cycles (e.g., see Blum, 1996) and 
components. 

Organization of modelling Activities and Mathematical Models 


MM structure consists of several stages. Blum and Leiss (2007) proposed to following: 
understanding the problem (constructing), simplifying (structuring), mathematizing, working 
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mathematically, interpreting, validating and exposing. Transitioning through these stages 
involves observations, measurements, interactions - described together as data, coding systems, 
methods of sampling, and data collection (Confrey & Maloney, 2007). The process of MM can be 
supported by various means with a real experimentation to be the most common and 
recommended (Thomas & Young, 2011). Since conducting real experiments is difficult in 
mathematics classrooms—that are traditionally not designed for that purpose—there is a need for 
trying other means, for example computerized experiments. Podolefsky, Perkins, and Adams 
(2010) proved that virtual experiments can substitute for real experiments in science, thus their 
adoption for enhancing MM has become more tangible in contemporary math classrooms. While 
interactive software may serve as a means of providing new mathematical insight, Alsina (2007) 
warned that it cannot replace learning by making, which implies that while using interactive 
software students need to be given opportunities to manipulate on the system variables and then 
discover the underlying principles by themselves. While progressing through stages of MM, the 
learners can achieve multifaceted cognitive goals and consequently increase their competencies 
in applying math in other disciplines. MM activities not only provide opportunities for 
constructing models but they also expand students’ views of mathematics by integrating it with 
other disciplines, especially sciences and engage students in the process of mathematization of 
real phenomena (Bleich, Ledford, Hawley, Polly, & Orrill, 2006). 

The MM processes usually conclude with a formulation of mathematical representations called 
models—that are themselves key artifacts of the modelling processes (Confrey and Maloney, 
2007). Elicited models are to be simplified, but accurate representations of some aspect of the 
real world (Winsberg, 2003). Models, can take various fonns, ranging from three- to one¬ 
dimensional physical objects, statistical expressions—mainly in forms of general linear 
models — to algebraic and differential equations, all of which symbolize system variables and 
model their behavior. “The generic purpose of constructing and making use of a model is to 
understand problems seen in a broad sense, encompassing not only practical problems but also 
problems of a more intellectual nature that aim at designing parts of the real world” (Niss et ah, 
2007, p. 8). 

Modelling at high school and college levels 

MM can be exercised at any school level, yet the search undertaking for the purpose of this meta¬ 
analysis revealed that the majority of the research concentrates on high school and college levels. 
MM on these levels focuses the learners on “learning mathematics so as to develop competency in 
applying mathematics and building mathematical models for areas and purposes that are basically 
extra-mathematical” (Nish et ah, 2007, p.5). Developing such competencies requires putting 
explicitly MM activities on the agenda of teaching and learning of mathematics. Research (Nish 
et ah, 2007) shows that there is no automatic transfer of learned mathematics concepts to being 
able to apply them in real-life situations. MM activities possessing exploratory character are to 
help students make the transfer more adaptable to their experiences. The content for exercising 
modelling depends on the schooling level. While at the secondary level, students are introduced 
to modelling dynamic phenomena, at the university courses, students are expected to be able to 
use calculus to model given situations and produce analytical results from analyzing their models 
(Alsina, 2007). At the university level, modelling activities often constitute a separate course 
aimed at training pre-service teachers (Lingefjard, 2007). In addition to acquiring competencies, 
MM activities at university level “open an excellent opportunity for revising the traditional 
assessment of course work and written examinations and go into the fruitful collection of good 
assessment practices” (Alsina, 2007, p. 472). At both levels; high school and college, the activity 
of modelling will require students to coordinate results of applied inquiry and construct and 
justify fonnulated models (Confrey & Maloney, 2007). 
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Prior Research Findings 

In supporting a need for this study, we searched for meta-analyses and other types of research 
syntheses on MM using ERIC (Ebsco), Educational Full Text (Wilson), Professional 
Development Collection, and ProQuest Educational Journals, as well as Science Direct and 
Google Scholar. Although several meta-analytic research studies aiming at various aspects of 
conceptualization of mathematical ideas were located, a meta-analysis specifically targeting 
research on MM or a synthesis of quantitative research was not found. A lack of such undertaking 
further supported a need for conducting this study. 

De kk ers and Donatti (1981) in their meta-analysis focused on computing the effect of using 
computer simulations as a medium for enhancing instructional strategies. The findings gathered 
from 93 empirical studies “did not support the contention that simulated activities cause an 
increase of students’ cognitive development (ES = - 0.075) when compared with other teaching 
strategies” (Dekkers & Donatti, 1981, p. 425). In light of these findings and to provide 
suggestions for further research, they suggested that “attention should be given to reporting 
details of methodology employed” (p. 426). The lack of promising results was associated with 
inadequate teaching methods that simulations were supposed to support. While summarizing 
effects of technology on creation of new environments for intellectual work in mathematics, Fey 
(1989) uncovered that technology, at that stage of development, was not helping students with 
graph interpretation, as was expected and suggested developing projects that will address and 
investigate eliminations of these difficulties. He also noted a need for a change in teachers’ 
perception regarding graph introduction—from teaching students “how to produce a graph to 
focusing more on explanations and elaboration on what the graph is saying ” (p. 250). Another 
advantage of using computers in math education is their capability of creating micro-worlds that 
allows students to make changes in their environments (Balacheff & Kaput, 1996). 

Quantification of learning effect sizes when the use of computer simulations were compared to 
traditional methods of learning was examined by Lee (1999), who meta-analyzed 19 empirical 
studies and concluded that they produced a moderate (ES = 0.54) learning effect size. Lee pointed 
out that “specific guidance in simulations helps students to perform better” (p. 81). In light of this 
finding, he advocated a need for placing more emphasis on the designing instructional support. A 
meta-analytic study conducted by Kulik (2003) who located 16 research studies published 
between 1990 and 2003 on the effectiveness of computerized exploratory environments in 
secondary schools revealed a moderate effect size of 0.32. This study did not provide further 
details on how the media of learning were embedded in the lesson cycles or discussed the design 
of instructional support. A substantial meta-analysis including studies published after 1990 was 
conducted by Li and Ma (2010) who extracted a total of 85 independent effect sizes from 46 
primary studies representing all grades from elementary to senior secondary school. These 
researchers computed the effect sizes of the impact of computer technology on mathematics 
education in K-12 classrooms. The overall effect size of ES = 0.28 supported the claim that using 
technology in mathematics classes improves students’ achievements. A corresponding subgroup 
moderator analysis revealed that effect of using simulations (ES = 1.32) outpaced the effect of 
tutorials (ES = 0.68). They also investigated moderator effects such as the type of learning 
environment and found out that “using technology in school settings where teachers practiced 
constructivist approach to teaching produced the larger effects on students’ mathematics 
achievement than using technology in school setting where teachers practices traditional teaching 
methods” (Li & Ma, 2010, p. 234). This finding supports Confrey’s and Maloney’s (2007) thesis 
that “knowledge should be subjected to criteria of functional fitness that is akin to the 
constructivist concept of viability” (p. 58). In a similar vein, Hussmann (2007) argued that 
technology can support to situate in constructivist paradigm two important mathematical 
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objectives; “function construction contributing to building ideas, and function iteration that 
initiates a change of concept” (p.348). Other researchers focused on investigating more specific 
constructs. Lege (2007) found out that having students formulate mathematical models and then 
having them validate the models generates higher learning effects as opposed to having students 
use prearranged models (formulas). He further claimed that the difference is accounted for the 
degree of ownership in model enacting: students who were involved in formulating the models 
varied given key assumptions and linked the keys together using selection criteria, whereas 
students in the control group passively constructed their models based on a single consideration. 
Lingefjard (2005) also concluded that after being immersed in MM activities, students handled 
word problems better than those taught by conventional methods. Research conducted by 
McBride and Silvennan (1991) revealed that MM used during integrated lessons increased 
students’ achievement in all involved subjects, not only math. 

Despite MM competencies being wider accepted as a part of mathematical literacy, modelling 
still faces unresolved issues that prevent the process of its conceptual framework design from 
solidifying. One such issue involves the stage of model validation. Zbiek and Conner (2006) 
suggested that students be given multiple opportunities to verify derived models. They also 
pointed out that MM supported only by pen-and-pencil might be lacking a reality aspect. Bleich 
and colleagues (2006) expressed concerns about inadequate teacher methodological preparation 
in inducing graphical representations of motion problems. A similar conclusion was reached by 
Sokolowski and Gonzalez y Gonzalez (2012) whose research revealed that teachers face 
obstacles in finding methodology that would help them guide the students through transitioning 
from observation to mathematization. 

In sum, the major meta-analyses along with other research reported positive learning effects when 
MM was applied to enhance math learning objectives. Yet, the information associated with the 
type of instructional support that appears to be of high significance along with the extent to which 
contents from other disciplines should be induced into modelling activities is limited. This 
synthesis has also revealed that there are also unanswered questions regarding instrumental 
implementation of this learning method. By undertaking this study, we attempted to fill in the gap. 
The purpose of this meta-analysis was to synthesize peered- reviewed quantitative research 
findings on MM at the high school and college (tertiary) levels and search for ways of advancing 
the techniques of developing students’ modelling competencies. Although “research in 
mathematics education has shown that the success of the modelling approach in mathematics at 
tertiary level does exist” (Alsina, 2007, p. 473), a study that would quantify the effect sizes of 
larger pool of research has not been yet undertaken. 

Research Methods 

We undertook a meta-analysis developed by Glass (1976) as a research method because meta¬ 
analysis helps to (a) to integrate the findings of individual research to fonnulate more general 
inferences about the effects of heuristic techniques applied during MM activities (b) to address 
some of the limitations of the previous research by allowing for construct formulations and 
evaluation and (c) evaluate effectiveness of MM activates using larger research pool since such a 
method has not been found in the prior literature. Zawojewski (2010) identified two types of 
research objectives on MM: (a) development and evaluation of the models formulated by learners, 
and (b) instructional tools and learning media applied during the modelling activities. This 
research intended to examine the findings of the former; effectiveness of instructional tools and 
learning media. Furthermore, through undertaking subgroup moderator analysis and identifying 
conditions that generate the highest learning effects, we hoped to also formulate suggestions for 
improving students’ performance on MM tasks. 
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Research Questions 

The fonnulation of the research questions was supported by (a) suggestions found in the prior 
literature, (b) development of contemporary views on the role of MM in school practice, and (c) 
the type of research methods employed. Intertwining these three venues, the following research 
questions were enacted: 

1. What are the magnitude and direction of the learning effect size when the learning is 
situated in MM as compared to conventional methods of learning? 

2. What are the moderators that affect students’ achievement during modelling activities and 
what are their effect sizes? 

Key Term Descriptions 

The literature search was guided by the following operational terms: mathematical modelling, 
model - eliciting activities, medium utilized for a model construction, mathematical model, 
experimental study, and effect size statistics. 

Mathematical modelling is a process of encountering a situation, problematizing it, and bringing 
inquiry, reasoning, and mathematical structures to transform the situation (Confrey and Maloney, 
2007). Literature showed that the tenn mathematical modelling can describe two types of activity: 
translating the real— world system into mathematical terms for the purpose of solving a problem 
or for the purpose of analyzing a situation by applying various steps associated with 
accomplishing that goal (Gravemeijer, 1997). MM of both of these types of activities will be 
included in this study. 

Model - eliciting activities (MEA) are defined as “problem solving activities that require students 
to express their current ways of thinking in forms that are tested and refined multiple times and 
that elicit a model” (Lesh & Yoon, 2007, p.162). In order to be termed MEA, an activity must 
satisfy six principles developed by Lesh and Kelly (2000): (1) the reality principle (2) the model 
construction principle (3) the self-assessment principle (4) the construct documentation principle 
(5) the model share-ability and reusability principle and (6) the simplicity principle. 

Medium utilized for a model construction is defined as a form of infonnation presented to the 
learners. The following are the possible media types: data tables, written text problems, 
computerized interactive simulations, or real experiments. 

Model was operationally defined as a mathematical construct designed and formulated to study a 
particular real-world system or phenomenon (Confrey, 2007). Mathematical models can include, 
but are not limited to graphical, symbolic, and physical representations. 

Experimental study is a type of research that seeks to determine whether an intervention had the 
intended casual effect on the participants. The following are the key components of an 
experimental study: (a) pre-posttest design (b) a treatment group and a control group and (c) 
random assignment of study participants (Shadish, Cook & Campbell, 2002). 

Effect size statistics (ES) is a measure of strength of an outcome after treatment, in a form of MM 
was applied. ES was used to quantify student achievement in each of the located studies. The 
magnitude of the effect size was calculated using Hedge’s (1992) formula. 
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g = — 1 --, where 
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x, represents the posttest mean score of the treatment group (situated in modelling environment) 


x 2 represents the posttest mean score of the control group (taught by traditional methods) 


5 = 


(n.-^SD; +(n 2 -\)SP{ 
n ] +n 2 - 2 


represents coupled standard deviation; where 


n ] , n 2 represent the sample sizes of the control and treatment groups respectively and 


SD } , .SIX represent standard deviations of treatment and control groups mean scores. 

Effect size will be expressed numerically in a decimal fonn. 

Data Collection Criteria 

The research included only peer-reviewed studies published in journals because such studies 
represented methodologically high quality research (Lipsey & Wilson, 2000). Although bias 
against null findings cannot be completely removed even in peer-reviewed journals (Cooper, 
2009) high quality research provides means of computing moderator effects that was also 
intended in this study. The initial search criteria was restricted to the following constrains: (a) 
time span which included papers published between January 1, 2000, and February 27, 2013; and 
(b) experimental research that provided means for calculating effect size statistics (c) level and 
subject of teaching; high school and college math courses and (d) MM as process to transfonn a 
situation into a model and analyze or solve it. The section that follows defines, in more details, 
descriptive and inferential parameters that were extracted from each study. 

Descriptive and Inferential Parameters 

Descriptive parameters encompassed the following: the grade level of the group under 
investigation, the locale where the studies were conducted, the sample size representing the 
number of subjects in experimental and control groups, the date of the study publication, the 
duration of the study, and the total time interval that the subjects were under treatment. The total 
treatment time was introduced due to a high diversity of treatment frequency; thus, for instance, if 
the study lasted 2 months and the treatment was applied twice a week for 3 hours each session, 
the reporting is depicted as 2 months/48 h. Inferential parameters included posttest mean scores 
of experimental and control groups along with their corresponding standard deviations. If these 
were not provided, F-ratios or t-statistics were recorded. Although most of the studies reported 
more than one effect size describing also other constructs than students’ achievement (see e.g., 
Schoen & Hirsch, 2003; Wang, Vaughn, & Liu, 2011), the current study focused on reporting 
effects of student achievement only. As experimental groups were under treatment of 
mathematical modelling, control groups were taught by traditional methods. 

Descriptions of groups and their classes 

A total of 14 classes were formulated and grouped according to their descriptive purposes in 
Table 1. The classes were used for coding purposes. 
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Table 1. Summary of Groups and Their Classes 


Group 

Classes 

Study general 
characteristics 

Research authors 

School level (high school or college) 

Subject area (calculus, statistics, algebra or geometry) 

Locale of the research (country where the study was conducted) 

Year of publication (year when research was published) 

Type of publication (peer-reviewed) 

Study methodological 
characteristics 

Instrumentation (computer-supported activity or pen and paper) 

Reliability of measure (researcher-developed instrument (local) or standardized tests) 

Type of research (quantitative) 

Group assignment (randomized or quasi-experimental) 

Sample size (number of participants in control and experimental groups) 

Study design 
characteristics 

Program used, research specifications (verbal descriptions) 

Duration of treatment (in semesters, weeks, or days) 

Frequency of treatment assignment (in hours per day or other metrics provided) 

Medium for model construction (computer or context provided on paper) 

Learning setting (student centered) 


In the process of collecting the research literature, ERIC (Ebsco), Educational Full Text (Wilson), 
Professional Development Collection, and ProQuest educational journals, as well as Science 
Direct, Google Scholar, and other resources available through the university library, were used. 
The initial search terms were defined reflecting formulation of classes focusing mainly on study 
methodological characteristics and design. The following terms were utilized to locate the 
relevant literature: mathematical modelling, model eliciting activities, simulations, computers in 
mathematics, mathematics education, student achievement, high school, college. These search 
criteria returned 241 articles. After a review, it was revealed that eight studies satisfied the 
inclusion criteria. Most of the rejected studies focused on examining formulated models in the 
professional fields of engineering or medicine. In order to increase the pool, a further search was 
undertaken with broader conceptual definitions. This search included auxiliary terms that were 
found in descriptions of mathematical modelling activities, such as investigations in mathematics, 
techniques of problem solving, exploratory learning in mathematics, and computerized 
animations and learning. These modifications returned 82 research papers. After an additional 
scrutiny, 5 studies were added to the pool. The validity of the coding and the extracted data was 
supported by a double research rating that constituted of two teams; the primary authors and 
another professional who reviewed extracted studies for their adherence to selection criteria. The 
double rating was applied at the initial and at the concluding stages of the study. Any 
discrepancies were resolved. 


Research Analysis 

A total of 13 primary studies were used in this meta-analysis with a total of 1,670 participants. 
We realized that to have the most accurate data along with most accurate inferences, the 
modelling activities would have to be coded according to the MEA principles as defined by Lesh 
and Kelly (2000). However, such extractions were not feasible, due to MEA principles not being 
converted into quantitative constructs in these studies. Table 2 summarizes the studies’ features. 

Table 2. General Characteristics of the Studies’ Features 
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Authors 

Date 

Locale 

RD 

SS 

School 

Level/ 

Subject 

Research 

Duration/ 

Frequency 

Learning 

Setting 

Medium 

of 

Learning 

Young, 

Ramsey, 
Georgiopoulos, 
Hagen, Geiger, 
Dagley-Falls, 
Islas, Lancey, 
Straney, Forde, 
& Bradbury 

2011 

USA 

QE 

265 

College/ 

Calculus 

1 semester 
lh/week 

SC 

Comp 

Wang, 

Vaughn, 

& Liu 

2011 

Taiwan 

QE 

123 

College/ 

Statistics 

1 semester 
NP 

SC 

Comp 

Voskoglou 
& Buckley 

2012 

Greece 

QE 

90 

College/ 

Calculus 

1 semester 
NP 

sc 

Comp 

Laakso, 

Myller, 

& Korhoncn 

2009 

Finland 

R 

75 

College/ 

Statistics 

2 weeks 
2h/week 

sc 

Comp 

Milanovic, 

Takaci, 

& Milajic 

2011 

Serbia 

QE 

50 

HS/ 

Calculus 

1 week 

4.5h 

sc 

Comp 

Baki, Kosa, 

& Guven 

2011 

Turkey 

R 

96 

College/ 

Geometry 

1 semester 

NP 

sc 

Comp 

Bos 

2009 

USA 

R 

95 

HS 

Algebra 

8 days 
55min/day 

sc 

Comp 

Mousoulides, 

Christou, 

& Sriraman 

2008 

Cyprus 

QE 

90 

HS 

Statistics 

and 

Geometry 

3 months 

3h 

sc 

Comp 

Schoen 
& Hirsch 

2003 

USA 

QE 

341 

HS 

Algebra 

1 semester 
NP 

sc 

PP 

Scheiter, 

Gcrjets, & 

Schuh 

2010 

Germany 

QE 

32 

HS 

Algebra 

1 session 

2h 

sc 

Comp 

Eysink, 
de Jong, 
Bcrthold, 
Kolloffel, 
Opfermann, & 
Wouters 

2009 

The 

Nether¬ 

lands 

and 

Germany 

QE 

272 

HS 

Probability 

1 week 

sc 

Comp 

Bahmaei 

2012 

Iran 

R 

60 

College/ 

Calculus 

1 semester 

15 sessions 

sc 

PP 

Baki & Guveli 

2008 

Turkey 

QE/ 

MS 

80 

HS 

Algebra 

1 semester 
NP 

sc 

Comp 


Note. R = randomized, QE = quasi-experimental, RD = research design, SC = student centered, MS = mixed methods, 
Comp = computer, PP = pen and pencil, HS = high school, SS = sample size, NP = not provided. 

The majority of the studies (9, or 70%) were designed as quasi-experimental, while 4 (30%) were 
randomized. The study durations ranged from 2 hours to 1 semester. The average sample size for 
the study pool was 123 participants, with the highest of 272 participant conducted by Eysink and 
colleagues (2009) and the lowest sample of 32 students in a study by Milovanovic and colleagues 
(2010). When categorized by school level, college and high school were uniformly represented, 
with six high school studies (or 46%) and seven college studies (or 54%). When categorized by 
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learning setting, all of the studies were student centered, meaning that students worked on 
deriving models for the given problems using the teachers’ expertise only when needed. Such 
organized MM activities “provided students with opportunities to discuss employed strategies 
with each other, explore alternative solution pathways, interpret and evaluate the reasonableness 
of arguments and solutions and explain both results and reasoning to others” (Antonius, Haines, 
Hojgaard, Niss, & Burkhardt, 2007, p. 295). Model formulation was supported by using 
computerized simulations in 11 (or 85%) of the studies; two studies (or 15%) used the traditional 
pen-and-pencil approach. 

Inferential Analysis 

The inferential analysis of this study pool was initially performed using SPSS 21 (Statistical 
Package for the Social Sciences) software. We used the program to verify the homogeneity of the 
study pool, as suggested by Cooper (2009) and calculated the effect size for each study using 
posttest means on experimental and control groups, as suggested by Lipsey and Wilson (2001). 
Such standardized individual effect sizes were then corrected for population bias, and weighted as 
suggested by Hedges (1992). After weighted effect sizes were computed, the overall mean effect 
size statistic along with homogeneity statistics for the entire pool was calculated. The calculated 
homogeneity statistics (Qt = 329.74, with df = 16, p < 0.01) indicated statistically significant 
variation of the effect sizes; thus, a random-effect model was adopted for further data analysis. 
These computations allowed for answering the first research question: 

What are the magnitude and direction of the learning effect size when the learning is situated in 
MM as compared to conventional methods of learning? 

The mean effect size for the 13 primary studies (14 primary effect sizes) was reported to have a 
magnitude of 0.69 (SE = 0.05). A 95% confidence interval around the overall mean —Ci owe r = 
0.59 and C up per = 0.79—supported its significance (Hunter & Schmidt, 1990). The numerical 
value of the effect size of 0.69 is classified by Lipsey and Wilson (2000) as of a moderate size. Its 
positive direction indicated that the score of an average student in the experimental groups —who 
used MM to enhance problem-solving techniques— was 0.69 of standard deviation above the 
score of a student in the control groups, who was taught the same processes using traditional 
methods of instruction. One can claim that MM activities that involve rethinking the nature of 
givens and patterns and allow students’ own ways of reasoning are more effective than traditional 
problem solving approaches that are characterized by Lesh and Yoon (2007) as getting from pre- 
mathematized givens to mathematical goals. Table 3 provides a summary of the individual effect 
sizes of the meta-analyzed studies along with their confidence intervals, standard errors, and 
general descriptions of the treatment and computer programs used as a medium for model 
eliciting. 

Table 3. Effect Sizes of Applying Mathematical Modelling in High School and College 


Study 

(First 

Author) 

ES 

SE 

95% Cl 

Lower 

Upper 

Reliability of Measure Used 

Program Used, Research Findings, 
Research Specifications 

Bos 

(2009) 

0.70 

0.21 

0.18 

1.01 

Standardized Texas state 
assessment. Kuder-Richardon 
formula 20 for reliability: 
fpret .80 and i'postt .90. 

TI Interactive Instructional 
environment. 

Young 

(2011) 

0.61 

0.13 

0.10 

1.09 

(UCF) university faculty 

Math Department tests. Inter¬ 
rater reliability: r pre t= .82 and 
Uostt — .92. 

Research modelling activities 
(Excel) supported by computer. 
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Baki 

(2011) 

0.81 

0.26 

0.09 

1.11 

PCVT test with KR-20 of 
t'pret • 8 — and fp re t .80 
(Branoff, 1998). 

Interactive geometry software. 

Young 

(2011) 

0.04 

0.13 

0.34 

0.85 

University (UCF) faculty 
Math Department tests. 
Reliability: r pre = .82 and 
r post = -92. 

Research modelling activities 
(Excel) supported by computer. 

Wang 

(2011) 

0.45 

0.26 

0.08 

1.11 

Researcher-developed 20- 
item test, Conbach’s a = .91. 

Developed dynamic computer 
program that modeled real 
situations to test hypothesis. 

Voskoglou 

(2009) 

0.49 

0.22 

0.17 

1.03 

Researcher-developed test 
graded by two faculty 
members. 

Contextualized differential 
equations using computer 
programs. 

Laakso 

(2009) 

0.61 

0.24 

0.12 

1.07 

Researcher-developed test. 

Trakla2 to have learners developed 
probability principles. 

Milanovic 

(2010) 

0.67 

0.29 

0.02 

1.18 

Researcher-developed test, 
items the same on both 
pretest and posttest. 

Developed simulated program to 
evaluate integrals. Used 
Macromedia Flash 10. 

Mousoulides 

(2008) 

0.31 

0.22 

0.17 

1.03 

Researcher-developed test. 

Researcher-designed activities 
aimed at various math model 
formulations. 

Schoen 

(2003) 

0.53 

0.11 

0.38 

0.81 

Standardized calculus 
readiness test items. 

Developed new math curriculum 
that focused on modelling. 

Schciter 

(2010) 

0.57 

0.36 

-0.14 

1.33 

Researcher-developed test 
aligned with Reed (1999) 
categorization. 

Computer programs to enhance 
modelling through animated 
situations. 

Eysink 

(2009) 

4.49 

0.12 

0.35 

0.84 

Researcher-developed 44- 
item test. Reliability was 
determined by Cranach’s 
a = .64 and a = .82. 

Different multimedia settings to 
investigate the effect on students’ 
math inquiry skills. 

Bahmaei 

(2012) 

1.84 

0.26 

0.07 

1.13 

Researcher-developed test 
items. 

Researcher-developed activities. 

Baki 

(2008) 

0.43 

0.23 

0.14 

1.05 

Researcher-developed test 
items with reliability 
of Tpostt .62. 

Web-based mathematics teaching 
material (WBMTM). 


Note. ES = effect size, SE = standard error. 


All meta-analyzed studies reported a positive effect size when MM activities were used. The 
highest effect size of ES = 4.49 was reported by Eysink and colleagues (2009), who investigated 
the effect of multimedia on students’ modelling skills, and the lowest of ES = 0.04 was reported 
by Wang and colleagues (2011), who investigated the effect of using computerized programs to 
model differential equations. In order to support reliability of the assessment instrument, several 
researchers (e.g., Wang et ah, 2011) applied the Crnobach’s a- coefficient or Kruder - 
Richardon’s formula 20 (KR20). A reliability coefficient of the assessment instrument was 
applied in six (or 46%) of the studies. Table 3 contains also additional infonnation provided by 
the primary researchers that distinguish the applied medium within the study pool. In the majority 
of the studies, the modelling activities were supported by researcher-developed contexts. In order 
to identify potential moderators, the studies were further aggregated (see Table 4). 

Analysis of Moderator Effects 

The process of computing subgroup effects allowed for uncovering moderators that optimized the 
magnitude of the effect size statistic and to answer the second research question: 

What are the moderators that affect students ’ achievement during modelling activities and what 
are their effect sizes? 
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A set of four moderators was identified: school level, type of medium used for MM, length of 
treatment, and mathematics content domain. This categorization resulted in 10 subgroups whose 
effects were individually computed and summarized in Table 4. 

Table 4. Summary of Subgroups and their Weighted Effect Sizes 


Moderators and Their Groups 

N 

ES 

SE 

95 % Cl 

Upper 

Lower 

Grade Level 

High school 

7 

0.94 

0.07 

0.79 

1.08 

College 

7 

0.45 

0.08 

0.30 

0.61 

Medium Supporting MEA 

Computer simulations 

12 

0.72 

0.06 

0.60 

0.85 

Pen and paper activities 

2 

0.68 

0.10 

0.48 

0.88 

Treatment Duration 

Semester 

8 

0.46 

0.06 

0.34 

0.59 

Shorter than one semester 

6 

1.31 

0.10 

0.11 

1.50 

Content Domain 

Algebra 

4 

0.73 

0.09 

0.55 

0.91 

Calculus 

5 

0.38 

0.09 

0.19 

0.56 

Probability and Statistics 

4 

3.11 

1.17 

3.11 

3.80 

Geometry 

1 

0.81 

0.26 

0.09 

1.11 


Note, N = number of participants, ES = effect size, SE = standard error. 

The mathematical calculations of the moderators followed Cooper (2009), who suggested giving 
more weight to effect sizes with larger sample populations. Calculation of corresponding 
confidence intervals and standard errors were also enacted. The following sections provide a 
more detailed discussion of the identified moderators and their effect on student achievement. 

The effect of the school level on student achievement 

This block was created to mediate the effect sizes of students’ achievement between the high 
school and college levels. Although it was intended to differentiate not only among high school 
grade levels but also among college majors, due to the limited pool of studies, this idea was 
aborted and two general group levels—high school and college—were fonnulated. The effect 
size showed differences; high school students reported a large effect size of ES = 0.94 (SE = 
0.07), versus college-level reporting a moderate effect size of ES = 0.45 (SE = 0.08). This 
evidences that that high school students benefit more by being involved in modelling activities 
than college-level students and the difference can be accounted for other mediators (silent in 
these studies), such as the difference in difficulty level of high school and college mathematics 
courses or better acquaintance of high school students with modern computerized modelling 
media. As modelling is a relatively new mathematics learning method, some college students 
might find it difficult to alter their habits of considering mathematics as a subject of drill and 
practice to a subject that provides a basis for hypothesizing, explorations and opportunities for 
genuine applications. We hypothesize that a prior experience with modelling at lower school 
level might also have an impact on students’ achievement at the college level. The data 
accumulated in the pool did not provide the basis for supporting such claim though. However, if 
the infonnation were available, an additional moderator could be fonnulated and further 
computations conducted. Developing modelling skills and techniques that require solving higher- 
order problems that involve analyzing and synthesizing knowledge of multiple subject areas 
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requires certain time and effort. It seems though that the sooner developing such skills is initiated 
and brought forth, the sooner the learner will become acquainted and benefit from them. 

The effect of medium used during MM activities on student achievement 

Two learning media—computer simulations and written pen-and-pencil activities—were 
identified in the gathered pool. Computers were used in 12 (or 86 %) of the studies to support 
modelling activities, and written pen-and-pencil methods were used in two of these studies (or 
14%). The learning effect size produced by simulations was higher (ES = 0.72, SE = 0.06) when 
compared to traditional pen-and-pencil activities (ES = 0.68, SE = 0.10), yet the difference was 
not that large. An advantage of computer simulations is their interactivity that allows to 
conveniently observing the system outputs due to manipulating on independent variable(s) 
(Scheiter et ah, 2010) and also verifying derived model. The other advantage of using technology 
is engaging the learners in a new level of creative discovery that places them “in a situation 
where they naturally raise the question before being shown the result” (Pead, Ralph, & Muller, 
2007, p. 315) which seemingly triggers revisions or consolidations of previously learned concepts. 
A word of caution must be placed here: the medium itself, as noted by Bos (2009), will not 
generate learning because concepts, principles, and ideas do not reside in physical materials or 
classroom activities but in what students actually do and experience. Kadijevich (2007) suggested 
to “view computers as tools that expand human mental function” (p. 352). As many students 
experience difficulties in transferring their knowledge from the mathematical world to real (e.g., 
see Crouch & Haines, 2004), technology according to Keune and Henning (2003), can help 
reduce such difficulties by enabling the students to concentrate and master the subtasks that cause 
the most difficulties in the transferring process. Careful inquiry planning coupled with 
availability of interactive media are the prerequisites for initiating students’ engagement and 
knowledge transfer. Research also shows (e.g., see Young et ah, 2011) that providing students 
with too detailed descriptions of procedures without letting them explore and discover relations 
on their own is not an effective inquiry and a balance between what students input should be and 
the degree of provided guidance needs to be established and controlled. A relatively low pool of 
located pen-and pencil research (2, 14%) suggests that such modelling medium is diminishing 
from the research and more sophisticated computerized environments are being used. 

The effect of the treatment length on student achievement 

Two different classes were formulated to answer this question: one semester and shorter than one 
semester. At the college level, some of the research was designed using a platfonn of a MM 
lasting one semester that examined the effects of modelling activities embedded during the course 
(e.g., Voskoglou & Buckley, 2012). The time span for other research was shorter ranging from a 
2 h (e.g., Scheiter et ah, 2010) to 3 months 3 h per week (e.g., Mousoulides et al., 2010). The 
effect size computation for this subgroup showed that shorter treatments produced a higher effect 
of students’ achievement (ES = 1.31, SE = 0.10) than longer (ES = 0.46, SE = 0.06). The large 
effect size of ES = 1.31 that resulted primarily from computing the effect sizes of control groups 
taught by tightly choreographed traditional teaching methods and experimental using MM, 
supports the claim that providing students with opportunities for modelling problems where they 
“filter, interpret, relate, organize, or synthesize information” (Lesh & Yoon, 2007, p.169) brings 
positive effects in their learning even if such activities are shorter. 

The effect mathematical content domain on student achievement 

Four different domains were formulated for this subgroup: algebra, calculus, probability and 
statistics, and geometry. The frequency of studies in each level was highly dispersed, ranging 
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from one study that examined modelling the concepts of geometry to five studies that focused on 
modelling calculus concepts. According to computations, probability and statistics, generated the 
highest effect size (ES = 3.11, SE = 1.17). The magnitude of this effect size was inevitably 
affected by an outlier of 4.49 (Eysink et ah, 2009). If this study were removed, the effect size 
would have been ES = 0.21, with SE = 0.17. Lakoma (2007) identified several steps of students’ 
natural reasoning that lead them to developing stochastic concepts: “exploring a situation 
involving randomness, fonnulating a problem, creating a local model of the phenomenon, 
analyzing the mathematical model in order to solve the problem and comparing solutions 
obtained using the model with results of observations of the random phenomenon” (p. 391). The 
high effect indicates that students leam the concepts of probability and statistics more effectively 
when the concepts are modeled. 

The concepts of algebra and its predominated domain — function analysis— produced a 
moderately high effect size of ES = 0.73. Mathematical functions are difficult for students 
because they can be perceived structurally, as objects, and operationally, as processes (Sfard, 
1991). Modelling challenges the students to connect objects and processes of various function 
representations, thus one can hypothesize that this is one of the reasons that modelling effectively 
supports function understanding. The lowest effect size of ES = 0.38, SE = 0.09) in this subgroup 
was attributed to activities involving calculus concepts. As calculus is driven by a high diversity 
of function representations, a closer look at this result and searching for ways of improving it 
seems indispensable. It is expected at the university level, that “students are able to use calculus 
to model situations and herby to be able to produce analytical results from analyzing models 
(Blomhoj & Jensen, 2007, p. 52). The task of model formulation, for instance differential 
equations (e.g., Milovanovic et al., 2011) that are typically initiated by identifying a rate of 
change are poised to have bifocal difficulties; they require a deep understanding of principles of 
the underlying contexts and familiarity with various structures of differential equations. 

As modelling is to promote better understanding of all involved subjects, not only its 
mathematical part, teachers need to be prepared to” help students clarify a real problem, generate 
and select variables, setup conditions appropriately and confidently [...] and promote positive 
affective contexts about mathematics and the problem domain” (Kadijevich, 2007, p. 349). 
Knowledge of sciences as well as of other academia is required to be possessed by mathematics 
teachers to successfully lead students through the process of identifying embedded principles. 
Calculus, as a study of change and accumulation, provides a wide range of sophisticated 
apparatus for inducing mathematical modelling activities, but it seems that heuristic techniques 
applied during modelling calculus concepts can be revisited to better reflect students’ needs and 
their experiences. While the focus of the current research is on applying MM to problem solving, 
attempts to use MM to model concept introduction such as, for example function differentiability, 
or the first fundamental theorem of calculus seem as suggestions for further research to improve 
the effects of MM on students achievement in calculus. It is hypothesized that introducing 
calculus concepts in contexts will simultaneously provide students with ample application 
examples thereby making the underlying theorems more related to students’ prior experiences. 

Summary and concluding remarks 

Accumulating all of the inferences, it is concluded that modelling activities generate positive 
learning effects when compared to traditional teaching methods at high school and college levels 
in any content domain. As “a number of empirical and international comparative studies indicate 
that applications and modelling are less significant in everyday school life in many countries” 
(Kaiser & Maafi, 2007, p. 99), this study provides a robust support for a wider implementation of 
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modelling to mathematical school practice as a support of students’ mathematical learning 
through modelling activities as well as the development of students’ modelling competencies. 

A subsequent moderator analysis revealed that among the collected pool of studies, the setting 
that produces the most optimal learning effects is short activities conducted at a high school level 
supported by computer simulations as a medium for modelling. While real-world contexts 
provide legitimate sources for introduction of MM activities (Niss et ah, 2007), more work needs 
to be done in bringing forth the heuristic techniques that MM has to offer such as “creating 
interplay between the real world and mathematics toward more realistic and less stereotyped 
problem situations and [...] changing teachers’ conceptions, beliefs and attitudes and change 
classroom culture by establishing new socio-mathematical norms” (Bonotto, 2007, p. 186). 

This study uncovered also several concerns regarding domain specific instructional support that 
would guide the students through reasoning and firm their MM competencies (e.g., see Lim et ah, 
2009, Mousoulides et al., 2008). In school practice “the activity of modelling is a way to 
highlight a mathematical topic” (Makar & Confrey, 2007, p. 490) rather than focus on inferences 
that derived models can provide. Such organized activities might not benefit the learners in fully 
because they support “realizing mathematics -by pinpointing out applications of ideas and skills 
that are introduced [...] not mathematizing reality" (Lesh & Yonn, 2007, p. 163). Since the goal 
of teaching mathematics today is to treat it as a language for communication and as a tool for 
predictions and explanations of reality (Freudenthal, 1983), there is a need to place more 
emphasis on the context and have the learner focus first on the underlined principles before 
attempting to translate it in a mathematical model. These concerns suggest pathways for further 
studies focusing on merging students reasoning skills learned in other subjects (e.g., sciences) in 
a unified comprehensive MM process. 

If mathematics is to be taught as a language of communications, the learners need to be provided 
with directions on how to relate behaviors of variables of a given scenario with properties of 
specific functions. For example, students would need to realize that periodic occurrences will 
most likely be modeled by sinusoidal functions for which to know period of occurrence along 
with a maximum or minimum value is required (e.g., see Sokolowski & Rackly, 2011) or that two 
dimensional motion requires consideration of parametric equations. Students’ familiarity with 
functions properties are important initial steps toward mastering modelling competencies but 
their skills to relate identified contexts principles with a corresponding function properties are 
anticipated to benefit the learner even more, especially at the high school and college levels. 

Other suggestions for further research, that materialized might focus on (a) investigating the 
effect of MM on eliminating students’ science and mathematics misconceptions and (b) the 
degree to which math modelling activities should be contextualized, whether MM should be 
limited to fonnulating mathematical representations or should it be perceived as a bridge linking 
mathematics with other academia and provide more opportunities for scientific investigations. 

We are aware of certain factors limiting the study findings, one of which was the number of 
located primary studies and the aim that focused only on quantitative. While we focused on 
collecting available peer-reviewed studies published in journals, we realized that perhaps opening 
the research to other types of reports such as books, dissertations, technical reports, unpublished 
manuscript, conference proceedings and the like would increase the pool and consequently the 
significance of study findings. Extending the search criteria is suggested as an avenue for a 
further study along with conducting a parallel meta-synthesis of qualitative research. Another 
factor justifying the limited count of studies is the virtue of modelling that is not widely exercised 
in school practice yet despite its proven positive effects. However as this research showed, its 
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popularity is gaining momentum; 46% of the collected studies (N = 6) were conducted within the 
past two years suggesting that a potential for a more comprehensive meta-analytic study exists. 
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